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Abstract 

We study in this paper a control problem in a space of random variables. We show that its Hamilton 
Jacobi Bellman equation is related to the Master equation in Mean field theory. P.L. Lions in mum 
introduced the Hilbert space of square integrable random variables as a natural space for writing the 
Master equation which appears in the mean field theory. W. Gangbo and A. Switch [ID] considered this 
type of equation in the space of probability measures equipped with the Wasserstein metric and use the 
concept of Wasserstein gradient. We compare the two approaches and provide some extension of the 
results of Gangbo and Switch. 

1 INTRODUCTION 

We study first an abstract control problem where the state is in a Hilbert space. We then show how this 
model applies when the Hilbert space is the space of square integrable random variables, and for certain 
forms of the cost functions. We see that it applies directly to the solution of the Master equation in Mean 
Field games theory. We compare our results with those of W. Gangbo and A. Switch m and show that 
the approach of the Hilbert space of square integrable random variables simplifies greatly the development. 
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2 AN ABSTRACT CONTROL PROBLEM 


2.1 SETTING OF THE PROBLEM 

We begin by defining an abstract control problem, without describing the application. We consider a Hilbert 
space 'H , whose elements are denoted by X. We identify 'H with its dual. The scalar product is denoted 
by ((,)) and the norm by ||.||. We then consider functionals J~(X) and J-t(X) which are continuously 
differentiable on T-L. The gradients DxJ~(X) and DxXt(X) are Lipschitz continuous 


\\DxHXi) ~ DxHX 2 )|| < c\\Xi - X 2 \\ (2.1) 

| \DxXt(Xi) ~ DxXt{X 2 )|| < c\\Xi - X 2 \\ 

To simplify notation, we shall also assume that 

WDxXmi \\DxX T m\<c (2.2) 

So we have 

\\D x X(X)\\<c(l + \\X\\) (2.3) 

and 

|^(X)|<G(1 + ||X|| 2 ), (2.4) 

where we denote by C a generic constant. The same estimates hold also for Xt(X). 

A control is a function v(s) which belongs to L 2 (0, T;H). We associate to a control v(.) the state X(s) 
satisfying 


V = «(*) (2-5) 

as 

X(t) = X 

We may write it as X\t(s ) to emphasize the initial conditions and even Xxt(s',v(.)) to emphasize the 
dependence in the control. The function A(s) belongs to the Sobolev space H 1 (t,T;'H). We then define the 
cost functional 
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(2.6) 


Jxt(v(.)) = ^ f* \ \v(s)\\ 2 ds + £ F(X(s))ds + Jr(X(T)) 

and the value function 


V(X,t) = inf JxM.)) (2.7) 

«(•) 

2.2 BELLMAN EQUATION 

We want to show the following 
Theorem 1. We assume k2.1\) . 112.2 1) and 


A > cT(l + T ) 


The value function is C 1 and satisfies the growth conditions 


( 2 . 8 ) 


|E(A,t)|<C(l + ||X|| 2 ) 

\\D x V(X,t)\\ < C( 1 + 11AT11), < C{ 1 + ||X|| 2 ) 


(2.9) 


where C is a generic constant. Moreover DxV(X,t ) and -— ^ ^ are Lipschitz continuous, more precisely 


\\DxViX\t 1 ) - D x V{X 2 ,t 2 )|| < CHA 1 - X 2 || + Cf 1 - t 2 1(1 + HA 1 !! + ||X 2 | 
,dV(X 1 ,t 1 ) dV{X 2 ,t 2 ). 


( 2 . 10 ) 


dt 


dt 


< ciia 1 -x 2 ||(i + ha 1 !! + ||a 2 ||) + c\t l - t 2 |(i + ha 1 !! 2 + ||x 2 || 2 ) 


It is the unique solution, satisfying conditions 112.9 1) and \2.1U\) of Bellman equation 


f -T" Dx v||« + *w-o 

V(X,T) = At(A) 


( 2 . 11 ) 


The control problem (12. 5 [I . (12.61) has a unique solution. 


Proof. We begin by studying the properties of the cost functional J X t{v(-))- We first claim that J\'t(u(.)) is 
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Gateaux differentiable in the space L 2 (t,T;H), for X,t fixed. Define X v (s) by 


dX v (s) 

ds 


v(s), X v (t) = X 


and Z v (s) by 


_dZ^s)_ = DxJ: ^ Xv ^ Zv{t) = DxFt ( Xv (T)) 


then we can prove easily that 


d[i 


Jxt(v(.) + A«K-))U=o = f ((Av(s) + Z v (s),v(s)))ds 


( 2 . 12 ) 


Let us prove that the functional Jxt(v(-)) is strictly convex. Let ui(.) and v 2 (.) in L 2 (t,T; 


%). We write 


Jxt(0v i(.) + (1 - 0)u 2 (.)) = Jxt(v i(.) + (1 - 6)(v 2 (.) - ui(.))) 

= Jxt(v i(.))+ [ ^-Jxt(vi(.) + n(l - 0)(v 2 {.) - vi(.))) dfj, 

Jo dfi 

From formula (12.121) we have also 

+ iiOv(.)) = of ((A (v(s) + n9v{.)) + Z v+fj9i j(s),v(s)))ds 

Therefore 


d_ 

dfi 


Jxt(v i(.) + /i( 1 - 9)(v 2 (.) - ^i(-))) dn = (1 - 9) 


((A(ui(s) +/r(l - 9)(v 2 (s) 


vi («))) 


d" V 2 (s) V\ 


(s) ))ds} 


Similarly we write 


dxt(9v 1 (.) + (1 - 8)v 2 (.)) = Jxt(v 2 (.) + 0(ui(.) - v 2 (.))) 
f 1 d 

= Jxt(v 2 (.)) + / —Jxt{v 2 {.) + n8(v 1 (.)-v 2 {.)))dn 
Jo a L 1 


and 
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/ Jfl Jxt ^ V2 ^ + ^(' Ul (-) -^ 2 (.)))d/x = oj ((A(v 2 (s) + /Z0(?;i(s) - U 2 (s))) + 

+ ^ 2+ ^(^- W2 )(s), ui(s) - «2(s) ))ds} 

We shall set -Zi(s) = Z v1+/x /i_q\/ V2 _ Vi \(s) and Z 2 (s) = ^ 2+/i 5 /(„ 1 _„ 2 )(s). Combining formulas, we can write 


Jxt(0v i(.) + (1 - 0)v 2 (.)) = 0</xt(?d(.)) + (1 - 0)</xt(v 2 (.))+ 


(2.13) 


+ 0 ( 1 - 0 ) 


A 

2 Jt 


I j^i(s) - v 2 (s)\\ 2 ds + [ d/i f {(Zi(s) - Z 2 (s),v 2 (s) - vi{s) ))ds 
Jt Jo Jt 


’0 Jt 

Let A'i(s) and X 2 (s) denote the states corresponding to the controls ui(.) + /i(\ — 9)(v 2 (.) — fi(-)) and 
v 2 {.) + n6{v\{.) — v 2 (.)). One checks easily that 


Xi(s) - X 2 (s) = (1 - /i) j ('Ui(ct) - v 2 (a))da 
and from the definition of Zi(.), Z 2 (.) we obtain 

ll^i(«) - < C[|| *i(T) - X 2 (T )II + [ T llX^a) - X 2 (a)\\d, 

J S 

and combining formulas, we can assert 

l|Zi(s) - Z 2 (s)\\ <c(l-/d)(l + T)J ||ui(a) -u 2 (<7)||do- 
Going back to (12.131) we obtain easily 


Jxt(0v i(.) + (1 - 9)v 2 (.)) < ej xt (v i(.)) + (1 - e)Jxt(v 2 (.))+ (2.14) 

_0(l__0)( A _ cr (l + T ))^ || Vl ( s ) - V2 (s)\\ 2 ds 

and from the assumption (12.81) we obtain immediately that Jxt(v(.)) is strictly convex. Next we write 

HX(s)) - X(X) = [\(D x X(X + 0 
Jo 


rs rs 

/ v(a)da), / v(a)da)) 

Jt Jt 


so , using m we obtain 
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|^(X( S )) - X(X)\ < c(l + ||X||)|| j\{a)da\\ + ||| J\(a)da\ 


< 


c 2 (l + ||X||) 2 c + 5 


25 


+ 


\J v{a)da\ 


for any 5 > 0. Using m we can assert that 


|J-(X( S ))| < C S ( 1 + ||X|| 2 ) + \\v(a)\\ 2 da 

A similar estimate holds for J~t{X[T)). Therefore, collecting results, we obtain 

| £X(X(s))ds + X t (X(T))\ < C S (1 + ||A'|| 2 )(1 + T) 

+^r(l + T)^' T || ? ;( S )|| 2 d S 

It follows that 


*«(»(■)) > A (c+OTi + r) £ | Ks) ||2 ds _ Cj(1 + || X ||2 )(1 + T) (2.15) 

Since A —cT(l + T) > 0,we can find 5 > 0 sufficiently small so that A —(c+5)T(l+ T) > 0. This implies that 
Jxt(v(-)) +°° as ff T ||v(s)|| 2 ds —> +oo. This property and the strict convexity imply that the functional 

,Jxt(v(-)) has a minimum which is unique. The Gateaux derivative must vanish at this minimum denoted 
by u(.). The corresponding state is denoted by Y(.). From formula (12.121) we obtain also the existence of a 
solution of the two-point boundary value problem 


dY 

ds 


Z(s) 

A 


-- = D X W)) 


Y(t) = X , Z(T) = D x X t (Y(T )) 


(2.16) 


and the optimal control u(.) is given by the formula 

n(s) = (2.17) 

In fact, the system (12.161) can be studied directly, and we can show directly that it has one and only one 
solution. We notice that it is a 2nd order differential equation, since 


6 













(2.18) 


d 2 Y 1 

w = x D ^ns)) 

( JY 1 

Y(t) = X — (T) = --D x F t (Y(T)) 

We can write also (12.181) as an integral equation 

Y(s) = X- S -X_±d x X t (Y(T)) -j£ T D x X(Y{a)){s Act — t)da (2.19) 

and we can view this equation as a fixed point equation in the space C°([t,T]\'H), namely Y(.) = JC(Y(.)), 
where 1C is defined by the right hand side of (12.191) . One can show that K, is a contraction, hence Y{.) is 
uniquely defined. Note also, that if we have a solution of (I2.16P and if u(.) is defined by (12.171) the control u(.) 
satisfies the necessary condition of optimality for the functional J X t(v(.)). Since this functional is convex, 
the necessary condition of optimality is also sufficient and thus u(.) is optimal. The value function is thus 
defined by the formula 


v (X,t) = \\Z{s)\\ 2 ds + j\{Y{s))ds + X t (Y(T)) (2.20) 

We now study the properties of the value function. We begin with the first property (12.91) . Using (12.151) we 
obtain 


V(X,t)>-C(l + \\X\\ 2 ) 

On the other hand, we have 


v(x,t) < j xt ( 0 ) = (T - 1)HX) + MX) < C( 1 + ||X|| 2 ) 

and the first estimate m is obtained. 

We proceed in getting estimates for the solution Y(.) of (12.191) . We write 


Using easy majorations, we obtain 


lin)ll= sup ||y(a)|| 

t<s<T 
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( 2 . 21 ) 


nn)ii< 

p(-)ii< 
IKON < 


||*||A + cT(T + l) 

A — cT(T + 1) 

A(i + r)c(i+ ||jr||) 

A - cT(T + 1) 

(i + r)c(i +1|*||) 

A - cT(T + 1) 


We then study how these functions depend on the pair *, t. We recall that Y(s) = Yxt(s). Let us consider 
two points *i,£i and * 2,^2 and denote Yi(s) = XxitiCs), ^(s) = Yx 2 t 2 ( s )- To fix ideas we assume t\ < £ 2 - 
For s > £2 we have 


Ti(«) - Y 2 (s) = *1 - *2 - j{D x Ft(Yi(T)) - D x F t (Y 2 {T))){s - £ 2 ) - jD x F T (Yi{T)){t 2 - £,) 

1 r T 1 r l2 

--T / {Dx^iY^a)) - D x J r {Y 2 {a)))(s/\a-t 2 )da - J / D x J~(Yi(cr))(s A cr — ti)dcr 

J t“2 J 


From which we obtain 


sup ||ri(s)-r2(a)||<||* 1 -* 2 || + Y T (i + r ) SU P ||Ti(s) -y 2 (s)||+ 

t 2 <s<T A t 2 <s<T 

+ t -^[\\DxMyi(m\ + [ T \\D X nyim\ds] 

Using the properties of D X Y and D x J~t and (12.2111 we can assert that 


sup ||Yi(s) -U 2 (s)|| < 

t 2 <s<T 


A 

cT{T + 1) 


(l|x,-x 2 || + fe- tl )(i + r)c x LUlM IJ 


More globally we can write 


sup l|y* ltl w-*Ws)|| < 

max(^i ,t2)<s<T 


A 

cT(T + 1) 


^||*i — * 2 || + |£ 2 — £i | (1 + T)c 


1 + max(||*i||, 11*21|) A 
A - cT{T + 1) J 

( 2 . 22 ) 


In particular 


sup ||U Yl t(s) - lx 2 t(s)|| < 

t<s<T 


A||*i-* 2 || 
A - cT(T + 1) 


Recalling that from the system (12.1611 we have 


(2.23) 
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Z(s) = f D x J r (Y(a))da + D x X t (Y{T )) 

J S 

and noting Z(s) = Z xt (s) we deduce from (|2.23|) that 


sup \\Z Xlt (s) - Z X2t (s)\\ < 

t<s<T 


c(T + l)A||Ad -X 2 1| 
A — cT(T + 1) 


(2.24) 


We next write 


JxAM-)) - JxAM •)) < v(Xi,t) - v(x 2 ,t) < J Xlt (M-)) ~ JxAM-)) 

where rti(.) and u 2 {-) are the optimal controls for the problems with initial conditions {X\,t) and (X 2 ,t), 
respectively. Denoting by Y Xl t( s ) an d X x 2 t(s ) the optimal states and by Y Xlt (s-, u 2 (-))i ^x 2 t(s; ui(.)) the 
trajectories ( not optimal) when the control tt 2 (-) is used with the initial conditions (A'i ,t) and when the 
control ui(.) is used with the initial conditions (X 2 ,t), we have 


Y Xlt (s-,u 2 (.)) - Y X2t {s) = Y Xlt (s) - Y Xat (s;ui(.)) = X l - X 2 


Therefore 

V(X u t) - V(X 2 ,t) < j\x(Y X2t (s) + X\— X 2 ) - F(Y X2t {s)))ds + T t (Y X 2t (T) + - X 2 ) - X T (Y X2t {T)) 

and by techniques already used it follows 

V{Xi,t) -V(X 2 ,t) < {{J T D x X(Y X2t {s))ds + D x F T (Y X2t (T)),X 1 -X 2 )) + ^(l + T)\\Xi-X 2 \\ 2 
which is in fact 


V{Xi,t) - V(X 2 ,t) < ((Z X2t (t),X 1 -X 2 )) + C -{l+T)\\Xi - A 2 || 2 (2.25) 

By interchanging the roles of X\ and X 2 we also obtain 

V{Xi,t) - V(X 2 ,t) > ((Z Xlt (t),X 1 -X 2 )) - C -{1 +T)\\Xi-X 2 \\ 2 (2.26) 

Using the estimate 1)2.241) we can also write 
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(2.27) 


V(X u t)-V(X 2 ,t) > ((Z X2t (t),X 1 - X 2 )) - c(T + 1)[ A _ ^ - X 2 || 2 

Combining (I2.25P and (j2.27|) we immediately get 


I V(X u t) - V(X 2 ,t) - ((Zx 2t (t),X 1 - X 2 ))\ < c(T + l)[ x _ cT X {T+ + ^ ] 11 X\ - X 2 || 2 (2.28) 

This shows immediately that V(X , t) is differentiable in X and that 

D x V(X,t ) = Z(t) = —Xu(t) (2.29) 

From the 2nd estimate (12.21 p we immediately obtain the 2nd estimate (12.91) . We continue with the derivative 
in t. We first write the optimality principle 

\ pt-\-6 pt-\-e 

V(X,t) = -j \\u{s)\\ 2 ds +J X(Y(s))ds + V{Y{t + e),t + e ) (2.30) 

which is a simple consequence of the definition of the value function and of the existence of an optimal 
control. From (|2.28D we can write 


V(X 2 ,t) - V(X u t) - az X2 t(t),x 2 -x 1 )) < C 11 X\ X 2 11 2 

where C is the constant appearing in the right hand side of (12.281) . We apply with X 2 = Y(t + e) , 
X\ = X.t = t + e. We note that Z Y (t+e),t+e(t + e) = Z X t(t + e) = —A u(t + e),since u(s) for t + e < s < t is 
optimal for the problem starting with initial conditions Y(t + e), t + e. Therefore 

rt-\-e 

V(Y(t + e), t + e) — V(X, t + e) < — X((u(t + e), / u(s)ds)) + C\\ / rt(s)ds|| 2 

Jt Jt 

Using this inequality in (12.301) yields 


\ pt-\-e rt-\-e rt-\-e rt-\-e 

V(X,t)— V(X,t + e) < — / ||n(s)|| 2 ds+ / J r (y(s))ds — X((u(t + e), / u(s)ds)) + C'|| / u(s)d, 

2 Jt Jt Jt Jt 


from which we obtain 
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lim inf 

e —>0 


V(X,t + e)-V(X,t) 

e 



Hx) 


(2.31) 


Next we have 


A /' ^ rt-\-e rt+e 

V(X,t + e)<— / ||ti(s)|| 2 d,s + / J r (y(s)— / u(a)da)ds + JFt(Y{T) — / u(a)da) 

2 Jt+e Jt+e Jt Jt 


therefore 


y(X,t + e) - 


\ /*£+e /'t+e 

*W) < ~2 J IK*)ll 2 <k- J HY(s))ds+ 

rT rt-\-e 

+ / (X(Y(s)- / u(a)dcr) - X(Y(s)))ds+ 

Jt+e Jt 

rt+e 

+Xt(Y(T) - j u(a)da) - ?t(Y(T)) 


and using assumptions on J 7 , J~t it follows that 


which means 


\ rt+e rt+e 

V(X,t + e)-V(X,t)<--j \\u{s)\\ 2 ds-X(Y(s))ds+ 

pT rt+e rt+e 

-((/ D x X(Y(s))ds + D x X T (Y(T)), u(a)da)) + -(1 + T)|| / «(a)da| 

Jt+e Jt Z Jt 

\ rt+e rt+e 

V(X,t + e)-V(X,t)<~- \\u(s)\\ 2 ds- JF{Y (s))ds+ 

rt+e rt+e 

+X((u(t + e),J u(cr)da)) +-(1 + T)\\ J u(a)dcr\\ 2 


We then obtain 


, V(X,t + e)-V(X,t) A ii/nm2 

lim sup ——--- X-LJ. < | | u (t) || 2 _ J-(X) 

e^o e 2 


(2.32) 


and comparing with (12.3111 we obtain immediately that V(X, t) is differentiable in t and the derivative is 
given by 


^(X,t) = ^\\u(t)\\ 2 - X(X) 


(2.33) 


Recalling (12.291) we see immediately that V(X,t) is solution of the HJB equation (12.111) . The 2nd estimate 
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(ESI) is an immediate consequence of the equation and the estimate on ||DxU(X, t)||. We next turn to check 
the addtional estimates (12.101) . We have 


DxV(Xi,ti) — D x V(X 2 , t 2 ) = Zxrtiih) - Z X2 t 2 (t 2 ) 


We assume t\ < t 2 then we can write 


ZxitAh) — Zx 2 t 2 (t 2 ) = [ (DxX(Yx 1 t 1 (s)) — DxJ r (Yx 2 t 2 (s)))ds+ 

Jti 

+D x J r T (Y Xl t 1 (T )) - D x X T (Y X2t2 (T)) (2.34) 

Using previously used majorations, we can check 


I \Z X iti (U) - Z X2 t 2 {t 2 )\\ < 


A c(T + 1) 

A — cT(T + 1) 


(||V, - V 2 || + 1 1 2 - tl |(l + r)c 1 + ;^g^ l Jf ll) ) (2.35) 


and the first estimate (12.101) follows immediately. The 2nd estimate (12.101) is a direct consequence of the 
HJB equation and of the first estimate (12.101) . So the value function has the regularity indicated in the 
statement and satisfies the HJB equation. Let us show that such a solution is necessarily unique. This is a 
consequence of the verification property. Indeed consider any control v(.) G L 2 (t,T;'H) and the state X(s) 
solution of (12.5p . Let V(x, t ) be a solution of the HJB equation which is C 1 and satisifies (12.91) . (12.101) . Then 
the function V(X(s),s) is differentiable and 


±V(X(s),s) = ^(X(s),s) + ((D x V(X(s),s),v(s))) = 

= - F(X(s)) + ^\\D x V(X(s),s)\\ 2 + ((D x V(X(s),s),v(s))) 
>-X(X(s)) - ^||u(s)|| 2 


from which we get immediately by inegration V(X,t) < J X t(v(.)). Now if we consider the equation 


= -± Dx V(X(s),s), X(t) = X 
as A 


(2.36) 


it has a unique solution, since D x V(X,s) is uniformly Lipschitz in X. If we set v(s) = ——DxV(X(s),s), 

A 

we see easily that V(X,t) = Jxt(v(-))- So V(X,t) coincides with the value function, and thus we have only 
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one possible solution. This completes the proof of the theorem. ■ 


□ 


3 THE MASTER EQUATION 

3.1 FURTHER REGULARITY ASSUMPTIONS. 

We now assume that 


J 7 , JrareU 2 (3.1) 

The operators D x J-(X).D x J-t{X) belong to According to the assumptions (12.11) we can assert 

that 


\\D 2 x X(X)\\, \\D 2 x Xt(X)\\<c (3.2) 

where the norm of the operators is the norm of £('H;'H). Recalling the equation (12.191) for U(s), we differ¬ 
entiate formally with respect to X to obtain 


D x Y(s) = I - S -^D 2 x X t {Y{T))D x Y{T) (3.3) 

D 2 x T(Y{a))D x Y{a)(s Aa- t)da 

so, D X Y(.) appears as the solution of a linear equation, and we see easily that it has one and only one 
solution verifying 



sup ||U x y(s)|| < -— x 
t<s<T X — cT(T + 1) 


(3.4) 


It is then easy to check that D x Y(s ) is indeed the gradient of Y X t(s) with respect to X, and the estimate 
is coherent with (12.101) . Since D x V(X,t) = Z{t ) = Z xt (t) with 


Z{t) = J T D x X(Y{s))ds + D x F t {Y{T )) 


we can differentiate to obtain 


n 2 

u x 


V(X,t) = j T D 2 x X(Y(s))D x Y(s)ds + D 2 x X T {Y(T))D x Y(T) 


(3.5) 
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and 


which is coherent with (12.241) . 


D 2 x V(X,t )|| < 


A c(T + 1) 

A - cT(T + 1) 


(3.6) 


3.2 MASTER EQUATION 

We obtain the Master equation, by simply differentiating the HJB equation (12.111) with respect to X. We 
set U(X,t) = DxV(X,t). We know from (12.211) that 


mx,t) ii < 


A(l + r)c(l + ||A||) 
A — cT(T + 1) 


(3.7) 


The function U (A, t) maps T~L x (0, T) into 77. From (13.61) we see that it is differentiable in A, with D X U{ A, t) : 
77 x (0 ,T) ->• £(77; W) and 


\\D x U(X,t)\\ < 


Ac(l + T) 

A — cT(T + 1) 


From the HJB equation we see that U(X,t) is differentiable in t and satisfies the equation 


(3.8) 


% - \D x U(X,t)U(X,t) +D x F{X) = 0 
ot A 

U{X,T) = DxXt(X) 


(3.9) 


We have the 

Proposition 2. We make the assumptions of Theoren[ T\ and hS.l\) . Then equation \3.9\) has one and only 
one solution satisfying the estimates i f A. 7)1 . i fA.Al) . 

Proof. We have only to prove uniqueness. Noting that 

D x U(X,t)U(X,t) = ^D x \\U(X,t)\\ 2 

we see immediately from the equation that U(X,t) is a gradient. So U(X,t) = D x V(X,t). Therefore (13.91) 
reads 
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D x(^-^\\ Dx V\\ 2 + H x )) = 0 

D x V(X,T) = D x F t {X) 


We thus can write 


% ~l x \\D X v\?+Hx) = m 

V(X,T) = D x X T (X) + h 

where f(t) is purely function of t and h is a constant. If we introduce the function </?(t) solution of 

^ = /(*)> ( f( T ) = h 

the function V ( X , t) — ip(t) is solution of the HJB equation (12.111) and satisfies the regularity properties of 
Theorem [0 From the uniqueness of the solution of the H JB equation we have V ( X , t) — (p(t) = V ( X, t ) the 
value function, hence U(X,t) == D x V(X,t), which proves the uniqueness.■ □ 


4 FUNCTIONALS ON PROBABILITY MEASURES 

4.1 GENERAL COMMENTS 

If we have a functional on probability measures, the idea , introduced by P.L. Lions mi, m is to consider 
it as a functional on random variables, whose probability laws are the probability measures. Nevertheless, it 
is possible to work with the space of probability measures directly, which is a metric space. The key issue is 
to define the concept of gradient. For the space of probability measures, it is the Wasserstein gradient. We 
shall see that, in fact, it is equivalent to the gradient in the sence of the Hilbert space of random variables. 


4.2 WASSERSTEIN GRADIENT 

We consider the space T 2 (R n ) of probability measures on R n , with second order moments, equipped with 
the Wasserstein metric ITA//, ;v),defined by 


wsW) 


inf [ 

7er JRn xR n 


\£,-v\ 2 1 (d£,dri) 


(4.1) 
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where r(/x, u) denotes the set of joint probability measures on R n x R n such that the marginals are // and v 
respectively. It is useful to consider a probability space P and random variables in T~L =L 2 (Q, A , P\ R n ). 
We then can write /x = Lx and 


W$(n,v) = inf E\X-Y\ 2 

x, Yen 

Lx = H 
Ly = V 

When the probability law has a density with respect to Lebesgue measure, say m(x) belonging to L l (R n ) 
and positive, we replace the law by its density. Note that f \x\ 2 m(x)dx < +oo. We call the 

space of functions / : R n —> R n such that f Rn \f(x)\ 2 m(x)dx < +oo. We consider functionals _F(/x) on 

Vo(R n ). If /x has a density m we write F(m). If m £ L 2 (R n ), we say that F(m ) has a Gateaux differential 

. . . dF(m) , . . 

at m, denoted by —-- (x) it we have 

dm K J 


F(m + 9fj) — F(m) 

lun-x- 

0 ->o 9 


I R n 


dF{m) 

dm. 


(x)/j,(x)dx, M/m £ L 2 (R n ) 


(4.2) 


d F(m) 

and ——-(x) £ L 2 {R n ). For probability densities, we shall extend this concept as follows. We say that 

dF(m) 


dm. 


dm. 

( x) £ L^R 71 ) is the functional derivative of F at m if for any sequence of probability densities m e 


3 F (in) 

in V 2 (R n ) such that Wo(m e ,m ) —> 0 then —--(.) eLi, ( R n ) and 

dm. e 


r dF(m) 

F(m e ) - t(m) - J Rn (x)(m e (x) - m{x))dx 

W2{m e , m) 


0, as e —y 0 


(4.3) 


3 F (m) 

The function —--(.) is called the functional derivative of F(m) at point m. Let us see the connection 

dm 

with the concept of Wasserstein gradient on the metric space V 2 (R n ) ■ We shall simply give the definition 
and the expression of the gradient. For a detailed theory, we refer to Otto [IB], Ambrosio- Gigli- Savare |U , 
Benamou-Brenier {2], Brenier p], Jordan-Kinderlehrer-Otto m, Otto m, Villani m 

The first concept is that of optimal transport map, also called Brenier’s map. Given a probability 
v £ V 2 {R n ),the Monge problem 


inf / \x — T(x)\ 2 m(x)dx 

{T(.)\T(.)m=u} J Rn 

has a unique solution which is a gradient T{x) = DQ(x). The notation T(.)m = v means that n is the image 
of the probability whose density is m. The optimal solution is the Brenier’s map. It is noted We do not 
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necessarily assume that u has a density. The following property holds 



x — DQ(x)\ 2 m(x)dx 


This motivates the definition of tangent space T(m) of the metric space T > 2 {R n ) at point m as 


(4.4) 


T{m) = G C™(R n )} 

We next consider curves on V 2 (R n ), defined by densities m(t) = m(t)(x) = m(x,t). The evolution of m(t) is 
defined by a velocity vector field v(t) = v(t)(x) = v(x,t) if m(x,t) is the solution of the continuity equation 


dm 


- 7 ^- + div (v(x, t)m(x, t)) = 0 


(4.5) 


rn(x. 0 ) = m(x) 


We can interpret this equation in the sense of distributions, and it is sufficient to assume that J 0 T f Rn \v{x, t)\ 2 m(x , t)dx 1 


+ 00 , VT < + 00 . This evolution model has a broad sprectrum and turns out to be equivalent to the property 
that m(t) is absolutely continuous in the sense 


W 2 (m(s), m(t)) < / p(a)da , Vs < t 


with p(.) locally L 2 . Now, for a given absolutely continuous curve m(t), the corresponding velocity field is 
not necessarily unique. We can define the velocity field with minimum norm, i.e. v(x, t ) solution of 


inf < / / \v(x,t)\ 2 m(x,t)dxdt 

{Jo J R n 


dm 


gj- + div (v(x, t)m(x, t)) =0 


(4.6) 


The Euler equation for this minimization problem is 


/ v(x,t).v(x,t)m(x,t)dt = 0, Vu(x,t)|div (v(x,t)m(x,t)) = 0 
Jr« 


= Ua.e. 


which implies immediately that v(t) G T(m(t)) a.e. t. Consequently, to a given absolutely continuous curve 
m(t) we can associate a unique velocity field v(t) in the tangent space T(m(t)) a.e. t. It is called the tangent 
vector field to the curve m(t). It can be expressed by the following formula 


T m(t+e), x _ 

r ±rn d) ^ x> 
v{x, t) = iim--- 


£->0 


(4.7) 
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the limit being understood in L 2 m ^{R n \ R n ). The function T™^+ e \x) is uniquely defined. Since by (4.3) , 
\\T™^ e \x) — x \\ L 2 ^ = W 2 (m(t),m(t + e)), we see that, for any absolutely continuous curve 

W 2 (m(t),m(t + e)) < C(t)e (4.8) 

In the definition of the functional derivative, see we can write 

F(m e ) — F(m) — f Rn \x)(m € (x) — m(x))dx 

-- > 0, as e ->• 0 (4.9) 

provided the sequence m e is absolutely continuous. 

Suppose that we consider the curve corrresponding to a gradient Z4<h(x) where 4>(x) is smooth with 
compact support, i.e the curve m(t) is defined by 


din 

+ div (D$(x)m(x,t)) = 0 
m(x, 0) = m(x) 


(4.10) 


Since it is a gradient, D&(x) has minimal norm and we can claim from (14.71) that 


D<&(x) = lim 


T% {e \ X ) -X t2 , D n. t~)TI\ 


e->0 


in Lf n (R n ] R n ) 


(4.11) 


We consider now a functional F(m) on Vo(R n ), and limit ourselves to densities. We say that F(m ) is 
differentiable at m if there exists a function r(.x, m) belonging to the tangent space T{m) with the property 


F(m(e)) — F(m) — f Rn Y{x,m).{Tm^ e \x) — x)m(x)dx 
W 2 (m, m(e)) 


0, cis 6 —y 0 


(4.12) 


We recall that , see ( 14.41) Il^m, m(e)) = \\Tm^ e \x)—x \\ L 2 . The function T(x, m) is called the Wasserstein 


gradient and denoted \7F m (m){x). If we apply this property to the map m(t) defined by (14.101) . this is 
equivalent to 


F(m(e)) — F{m) 


I R n 


r(®, m) ,D&(x)m(x)dx 


From the continuity equation (14.101) . using the regularity of 4>, we can state that 


m(x, e) — m(x) 


—div (D&(x)m(x)), as e —> 0, in the sense of distributions 
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If F(m) has a functional derivative we obtain 


Therefore we obtain 


F(m(e)) — F(m) 
e 



dF{m) 

dm 


(x)div ( D&(x)m(x))dx 



x, m).D&(x)m(x)dx 


f dF(m) 

Jr n dm 

r D dF{m) 

I Rn dm 


(x)div (D&(x)m(x))dx 
(x) .D$>(x)m(x)dx 


If we assume that D (x) € L^(I? n ; R n ), we can replace -D<f>(x) by any element of T(m). Since T(x, m ) 
and D dF Q™^ (x) belong to T(m), it follows that 


VF m (rn)(x) = D^J-(x) (4.13) 

So the Wasserstein gradient is simply the gradient of the functional derivative. 

Remark 3. The concept of functional derivative, defined in (14. 3 j) uses a sequence of probability densities 
m e —> m, so it is not equivalent to the concept of Gateaux differential in the space L 2 (R n ), which requires 
to remove the assumptions of positivity and f Rn m{x)dx = 1. We will develop the differences in examples in 
which explicit formulas are available, see section El 


4.3 GRADIENT IN THE HILBERT SPACE H. 

The functional F(m) can now be written as a functional F(X) on R , with m = Cx■ We assume that 
random variables with densities form a dense subspace of R. Consider a random variable Y € R and let 
7 T (x,y) be the joint probability density on R n x R n of the pair (X,Y). So m(x ) = f Rn ir(x,y)dy. Consider 
then the random variable X + tY. Its probability density is given by 



ty,y)dy 


and it sastisfies the continuity equation 


dm 

~dt 


We have F(X + tY) = F(m(t)). Next 



ty,y)ydy ) 
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r F(X + iY)- F(X) 

lim- 

t —>-0 t 


((D x F(X),Y)) 


and 


. F(m(t )) — F(m) 

lim- 

t—y t 


dF(m) 


J Rn dm, 
f jj9F(m] 
l R n dm, 


(x)div ( / vr (x,y)ydy)dx 
J R n 

(x)-(f ir(x,y)ydy)dx 


I R" 




Thus necessarily 


D ^r {X) = VF ^( m )( X ) = DxHX) (4.14) 

So, the gradient in R reduces to the Wasserstein gradient, in which the argument is replaced with the 
random variable. In the sequel, we will use the gradient in T-L. 


5 MEAN FIELD TYPE CONTROL PROBLEM 

5.1 PRELIMINARIES 

Consider a function f(x,m) defined on R n x V- 2 {R n )- As usual we consider only m which are densities of 
probability measures, and use also the notation f(x,£ x ). We then define F(X) = Ef(X,£ x ). This implies 

F(X) = <&(m) = j f(x,m)m(x)dx (5.1) 

J R n 

We next consider the functional derivative 

d (x) = F (x, m) = f(x, m) + J^(£, m)(x)m(^)d^ (5.2) 

and we have 


We make the assumptions 


D x F(X)=D x F(XXx) 


(5.3) 
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(5.4) 


\D x F(x,m)\ < |(1 + \x\ + (j |f| 2 m(£)df)2) 

|-D x E(xi,mi) - -D x F(x 2 ,ro 2 )| < -(|xi - x 2 | + W 2 (mi,m 2 )) (5.5) 

which implies immediately the properties (12.11) . (12.21) . 

5.2 EXAMPLES 

We consider first quadratic functionals. We use the notation x = f Rn xm(x)dx. We then consider 

f(x, m) = — (x — Sx)*Q(x — Sx) + -x*Qx (5-6) 

then assuming that f Rn m{x)dx = 1, i.e. m is a probability density we have 

F(x, m) = -x*(Q + Q)x + X -x*S*QSx - x*(QS + S*Q - S*QS)x (5.7) 

D x F(x , m) = (Q + Q)x — (QS + — S*QS)x (5.8) 

We see easily that assumptions (I5.4IMI5.5I) are satisfied. 

We can give an additonal example 

f(x,m) = \ f K(x,£)m(£)d£, (5.9) 

z JR n 

with K(x,£) = K(£,x) and 

|A'(xi,6) - K{x 2 ,&)\ < C{ 1 + |xi| + |x 2 | + |6| + |6|)(|xi - x 2 | + |6 - 61) (5-10) 

\D x K(xi,£i) - D x K(x 2 ,&) I < |(|ah - x 2 \ + |6 - 61) (5-11) 

\D x K(0, 0)| < | 

We have 
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J^(£.™)( *) = = \ K i x ^) 

hence f Rn J^(£, m)(x)m(£)d£ = f(x,m) which implies 

F(x, m) = 2f(x, m) = I K(x,£)m(£)d£ (5.12) 

J R n 

We thus have 

|A c F(x,m)|| < [ \D x K(x,£)\m(£)d£ < 

J R n 

< |(1 + M + J \t,\m(Qd£) < 

< |(1 + \ x \ + (f |C| 2 m(f)dO*) 

If we take 2 densities mi, m 2 , we may consider 2 random variables Si,S 2 with the probabilities mi, m 2 . 
Therefore 


|D 3; F(a;i,mi) - D x F(x 2 , m 2 )| < | J D x (K(x i,£) - Ar(x 2 ,£))mi(£)d£||+ 

+|A J D x (Ji(x 2 ,Si)-A>2,H 2 ))| < 

|l*i - *21 + - e 2 | 2 

and since Ei,E 2 are arbitrary, with marginals mi, m 2 we can write (15.51) . In the sequel we also consider a 
functional /i(x, m) with exactly the same properties as / and write 


/» 

Fr(x,m) = h(x,m) + / —— (£,m)(x)m(£)d£ (5.13) 

Ft{X) = / h(x,m)m(x)dx, DxFt(X) = D x Ft(X, Cx) 

J R n 

5.3 MEAN FIELD TYPE CONTROL PROBLEM 

We can formulate the following mean field type control problem. Let us consider a dynamical system in R n 
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— = r(./-(.s). .s) 
x{t) = £ 


(5.14) 


where v{x, s ) is a feedback to be optimized. The initial condition is a random variable with probability 
density m(x).The Fokker-Planck equation of the evolution of the density is 


dm 

ds 


+ div(w(x)m) = 0 


(5.15) 


m(x, t) = m(x) 


We denote the solution by m v n(x,s). Similarly we call the solution of 115.1411 x(s;w(.)). We then consider 


the cost functional 


Jm,t{ v (-)) = T f f \v(x,s)\ 2 m v (\(x,s)dxds+ [ [ m v n(x,s)f(x,m v r)(s))dxds+ (5.16) 

z Jt JR n Jt JR n 

J R n 


which is equivalent to the expression 


Jm,t( v (-)) = \ J E\v(x(s-,v{.)))\ 2 ds + ^ Ef(x(s;v(.)),£ x {s))ds +Eh(x(T; v(.)), C X ( T )) (5-17) 

This is a standard mean field type control problem, not a mean field game. In [3] we have associated to it 
a coupled system of HJB and FP equations, see p. 18, which reads here 


~S + 2A \ Du \ 2 = F (xMs)) 

u(x,T) = Fr(x,m(s)) 
dm, 1 

—-—div( Dum) = 0 

ds A 

m(x, t) = m(x) 


(5.18) 


23 




This system expresses a necessary condition of optimality. The function u(x, t ) is not a value function, but 
an adjoint variable to the optimal state, which is m{x , s). The optimal feedback is given by 

v(x,s) = — — Du(x,s) 

A 

We proceed formally, although we shall be able to give an explicit solution of this system 
optimal feedback, then the value function V(m,t) = J m ,t(v(-)) is given by 

V(m,t) = — I I m(x, s)\Du(x, s)\ 2 dxds + f f m(x,s)f(x,m(s))dxds+ (5.20) 

J t Jru J t J R n 

+ / m(x,T)h(x,m(T))dx 
J R n 

The value function is solution of Bellman equation, see 01, M: written formally (it will be justified later) 


(5.19) 


If v(x, s ) is the 


dV 

1 W 



D dV(m,t) 
^ dm 


( 01 2 m(Qd£ + 



f(Z,m)m(£)d£ = 0 


V(m,T) 



h(£, m)m{£ > )d£, 


(5.21) 


5.4 SCALAR MASTER EQUATION 


We derive the master equation, by considering the function 


U(x, m, t) 


dV (m, t) 
dm 


(*) 


and we note that 


dU. 


d 2 V(m,t ) 
dm 2 


(®,0 


therefore the function is symmetric in x, £ which means 


dU, dU . 


By differentiating (15.2111 in m, and using the symmetry property, we obtain the equation 
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(5.22) 


dU_ 

~dt 


1 

A 




D ^rr ^ X ' m ’ 

--^\D x U( x ,m,t)\ 2 + F(x,m ) = 0 


£/(x, m, T ) = Ft(x, m) 


This function allows to uncouple the system of HJB-FP equations, given in (15.181) . Indeed, we first solve 
the FP equation, replacing u by U, i.e. 


r)rn I 

—-—di v(DU m) = 0 

os A 

m(x, t) = m(x) 


(5.23) 


then u(x,s) = U(x,m(s), s ) is solution of the HJB equation (15.181) . as easily checked. In particular , we 
have 


u(x, t) = U ( x , m, t) 


(5.24) 


5.5 VECTOR MASTER EQUATION 

We next consider U(x,m,t) = D x U(x,m,t). Differentiating (15.221) we can write 


dU_ 

~dt 


[ rn. i)(£) Zl(£, m, t)m(£)d£, — —D x U(x , m, t)U{x , m, t) + D x F(x.m) = 0 

I ftri OTfl A 


(5.25) 


U(x,m,T ) = D x FT(x.m) 


6 CONTROL PROBLEM IN THE SPACE U. 

6.1 FORMULATION 


If we set 


25 








F(X)=Ef(X,£ x ) 


f(x, m)m{x)dx 


( 6 . 1 ) 


X T (X) = Eh(XXx) 


h(x, m)m{x)dx 


F(x,m ) = f(x,m)+ [ ^ (x)m(£)d£ (6.2) 

J dm 

Fxix^m ) = h(x,m ) + [ m ) f x )m(£)d£ 

J am, 

We have 

D x X(X) = D x F(X,Ex) (6.3) 

D x Et(X) = D x F T (X,Cx ) 

We assume that 


lAr-FXxijmi) - .D x .F(2; 2 ,m 2 )| < -(|xi 


|-Dx-Pr(a:i,mi) - D x F T (x 2 , m 2 )| < -(|xi 


x 2 | + W 2 (mi,m 2 )) 
x 2 ] + W 2 (mi,m 2 )) 


(6.4) 


\D x F(x,m)\ < |(1 + |x| + |£| 2 m(£)d£) (6.5) 

|-D*-Fr(®,m)| < |(1 + |x| + ^ |£| 2 m(£)d£) 

It follows that 


II DxHXi) ~ D x F(X 2 ) || < || D x F(X 1 ,£ Xl ) ~ D x F(X 2 , £ Xl )\\+ 

+\\D x F(X 2 ,F Xl ) -D x F(X 2 ,£x 2 )\\ 
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<c\\X 1 -X 2 \\ 


and similar estimate for Ft -Therefore the set up of section IXT1 is satisfied. We can reinterpret the problem 
(15.141) , (15.171) or (15.151) , (15.161) as (12.51) , (12.61) which has been completely solved in Theorem [T| We shall study 
the solution of the abstract setting. Of course, the initial state X has probability law Lx = m. 

6.2 INTERPRETATION OF THE SOLUTION 

The key point of the proof of Theorem [T) is the study of the system (12.161) which has one and only one 
solution. We proceed formally. Consider the HJB-FP system (I5.18[) . The initial conditions are the pair 
(m,t), so we can write the solution as u m: t(x,s), m mi t(x,s). We introduce the differential equation 

^ = ~j Du (y( s ),s) ( 6 . 6 ) 

y(t) = x 

The solution ( if it exists) can be written y xm t(s ).Now let us set z xm t(s) = Du m t(y X mt(s), s). Differentiating 

dz 

the HJB equation (15.181) and computing the derivative — we obtain 


= D x F(y(s),m{s)) (6.7) 

z(T) = D x F T (y(T),m(T )) 

Now, from the definition of m(s) solution of the FP equation, we can write 

m{s) = y m t(s)(.)(m) (6.8) 

in which we have used the notation y m t(s)(x) = y m t{x,s) = y xm t(s) and y m t(s)(.)(tn) means the image 
measure of m by the map y m t(s)(.). So we can write the system (16.61) . (16.71) as 


= \ D xF(y(s),y(s)(.)(m)) 
y(t) = x C f s (T) = -jD x F T (y(T),y(T)(.)(m)) 


(6.9) 
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This is also written in integral form 


y(s) = x- ^D x F T (y(T),y(T)(.)(m)) (6.10) 

D x F(y(a),y(a)(.)(m))(s A a - t)da 

Now if we take yx,c x ,t( s )i then y(s)(.)(Cx) = £y(s) ■ Writing yx,c x ,t ( s ) = Y(s) to emphasize that we are 
dealing with a random variable, we can write (I6.10|) as 



Y(s) = X - ^ D x F t (Y(T),C y{T )) 
~\j MhW,A'( ff ))(sA(T- 


t)da 


( 6 . 11 ) 


which is nothing else than (12.61) recalling the values of DxJ~(X), DxFt(X), cf (16.31) . We know from Theorem 
HJliat (16.111) has one and only one solution in C°([t, T]; T~i) and in fact in C 2 ([t , T\]'H). This result, of course, 
does not allow to go from (16.111) to (16.101) , but it easy to mimic the proof. We state the result in the following 


Proposition 4. We assume \6-4\) , 1 (>. 5\) and condition \2.8\) . For given m,t there exists one and only one 
solution y m t(x,s) of Ui.lCh) in the space C(t,T; L‘^ n (R n ; R n )). 


Proof. We use a fixed point argument. We define a map from C(t,T; L^ n (R n ; R n )) to itself. Let z(x,s) a 
function in C(t, T; L^(i? n ; R n )). We define 


We have 


C(x,s) = x - ?-^D x F T (z(x,T),z{T)(.)(m))- 


D x F(z(x, a), z((i)(.)(m))(s A o — t)da 


|C(x,s)| < |x| + y ^(1 + \z(x,T)\ + ( f \z(£,T)\ 2 m(£)dt,) 1/2 )+ 

A Z J R n 

+ ^ f t (1 + \z(x,<t)\ + {J \z(£,a)\ 2 m(Z)d£,) 1/2 )da 
hence , from norm properties 
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Tc 


/ |C(x, s)\ 2 m(x)dx < , / / |x| 2 ra(x)dx + --(1 + 2( / |z(£, T)| 2 m(£)d,$;) 1 / 2 )+ 

I R n V A ^ ifl" 

+ ^ft ^ 1 + 2 ^ l R n l Z (^ a )| 2m (^)^) 1/2 ) d(7 


and we conclude easily that C belongs to C(t, T; L^ n (R n ] R n )). We set C = T(z). Using the assumptions and 
similar estimates, one checks that T is a contraction. We prove indeed that 


\\T{zi) - T(z2)\\c(t,T-,Ll l ) < (1---)lkl - Z 2 WC(t,T-,L 2 m ) 


( 6 . 12 ) 

□ 


It follows immediately that the solution y xm ,t(s ) = y m t(x,s ) satisfies the estimate 


Wymti^WcRT;^) < 


\yjf Rn |x| 2 m(x)dx + cT(T + 1) 


A - cT(T + 1) 

Since Yxt(s) = yx,c x ,t(s ) we deduce the first estimate (|2.2ip . We consider next 


(6.13) 


Zxmt(s) = z mt (x,s ) = D x F T (y(x,T),y(T)(.)(m))+ 
+ J D x F(y(x,a),y(a)(.)(m))da 


and from the assumption (16.51) we obtain easily 


\\Zmt{.-)\\c{t,T-,L 2 rn ) < C (1 + ^)(1 + I \ymt(-')]\c(t,T;L-* n )') 

hence 


(6.14) 


ll 2 mt(-)llcd,T;L2 i ) < ^ c (l + T) 



x| 2 m(x)dx + 1 


A - cT(T + 1) 


Clearly Z(s) = Zxt(s) = zx,c x ,t(s), see (12.161) . and we recover the 2nd estimate (12.211) . 
We can give more properies on y xm t(s). We write first 


(6.15) 
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y m t(xi,s) - y m t(x 2, s) = X 1-X2- 


s — t 


(D x F T (y mt (x 1 ,T),y mt (T)(.)(m)) - D x F T (y mt (x 2 , T), y mf (T)(.)(m)))- 


1 f 

-- / (D x F(y mt (xi,a),y mt ((T)(.)(m)) - D x F(y m t(x 2 ,a),y m t(a)(.)(m)))(s A a - t)da 


From (16.41) we obtain easily 


sup \y m t(xi,s) - y m t{x 2 , s ) < 

t<s<T 


A|xi - x 2 | 

A-cT(r+l) 


Also 


|s| + 

sup \y mt (x,s)\ < A- 

t<s<T 


Tc(l + T)(l + ^Jf Rn \tfm(Z)dZ) 
A - cT(T + 1) 

A — cT(T + 1) 


(6.16) 


(6.17) 


A similar estimate holds for sup t<s<T | z m t(x, s)|. 


7 BELLMAN EQUATION AND MASTER EQUATION 

7.1 THE VALUE FUNCTION 

The value function of the control problem in "H is given by 

V(X,t) = ^ £\\Z(s)\\ 2 ds + £ F(Y(s))ds + F t (Y(T )) 

in which Y(s) =yx,c x ,t( s ) an d Z(s) = Zx,c x ,t( s )- From this representation and the definition of F and Ft 
we can assert that V(X,t) depends only on Lx and thus can be written V(rn,t) with 


V{m,t) = ^~-f [ \z xmt {s)\ 2 m(x)dxds + [ [ f(y X mt(s),y m t(s)(.)(m))m(x)dxds+ (7.1) 

ZA Jt J R n Jt J R n 

+ / h(y xmt (T),y mt (T)(.)(rn))m(x)dx 
J R n 


From (16.151) we have 



Izxmtis^m^dxds < 


TA 2 c 2 (1 +T) 2 (1 + f Rn \x\ 2 m(x)dx) 
(A — cT(T + l)) 2 


and |U(T(s))| < C(1 + ||y(s)|| 2 ), therefore 
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I R n 


f(yxmt(s),y mt (s)(.)(m))m(x)dx\ < C(1 + / \y xmt {s)\ 2 m(x)dx ) 


and from the estimate (16.131) we obtain 


[ [ f(yxmt(s),y mt {s)(.)(m))m(x)dxds\ < CT[1 + 

Jt ■) R n 


A 2 f Rn \x\ 2 m(x)dx + T 2 (T + l) 5 


(A - cT(T + l)) 2 J 

and the third term in the right hand side of (ED satisfies a similar estimate. We thus have obtained 


\V(m, f)| < C(1 + / \x\ 2 m(x)dx) 
J R n 


(7.2) 


which is, of course, equivalent to the 1st estimate (12.91) . 


dV(rn, t ) 

We turn now to U(x,m,t) = —-—-— (x). We have seen formally in (15.241) that Uix^m^t) = u(x,t) = 

dm 


u m t(x,t). We need to prove it. We begin by giving a solution to the system HJB-FP equations (|5.18l) . We 
have the 

Lemma 5. We make the assumptions of Proposition [7} We can give an explicit formula to the system 
\5.18\) . We have 


1 f T f T 

Umt(x,t) = — J \z xmt {s)\ 2 ds + J F(y xmt (s),y mt (s)(.)(mf)ds+ 
+ F T (y xmt (T),y mt (T)(.)(m)) 


(7.3) 


and m m t(s ) = y mt (s)(.)(m). 

Proof. Indeed, if we look at F(x,m(s)) and i ? 7 ’(x,m(T)) in which m(.) is frozen, the HJB equation appears 
as a standard one for a deterministic control problem. This problem is simply 

dx 

* = v{s) 
x{t) = x 

A fT fT 

J x t(v(.)) = - J \v{s)\ 2 ds + J F(x(s),m(s))ds + F t (x(T), m(T)) 

in which the function m(s) is frozen, but not arbitrary. It is the function solution of the FP equation, in 
the system (15.181) If we write the necessary conditions of optimality, one checks easily that in view of the 
specific value of m(s),the optimal state is y X mt{s ) and the optimal control is --^z xm t{s). In plugging these 
values in the cost function, we obtain formula (17.31) . ■ □ 
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We may assume that 


\F(x, m)\, \Ft(x, m)\ < C(l + \x\ 2 + J |£| 2 m(£)d£) 


(7.4) 


We shall also assume that 


' SF(l ' m) -(a i^^(oi<£7(i + w , +ki , + / 


dm 


(7.5) 


m 71 dF(x,m )„,, / ^ 

|D " D{ 9m (5,I£C 


(7.6) 


We also make an assumption which simplifies proofs, but which can be overcome, with technical difficulties. 


/ (F(x, mi) — -F(:r, m 2 ){m\{x) — m 2 (x))dx > 0 
J R n 

(Fr(x,mi) — Fr(x,m2)(mi(x) — m2{x))dx > 0 


(7.7) 


This assumption allows to obtain the following interesting in itself result 

Proposition 6. We assume 17.71) . Then considering the system of HJB-FP equations \5.18 1) with initial 
conditions m\{x) and m 2 (x) and calling u\(x, s), m.\(x, s), respectively 772 ( 3 ;, s), 777 - 2 ( 3 ;, s) the solutions, we 
have the property 


i R n 


(u\(x, t) — U 2 (x, t))(mi(x) — m 2 {x))dx > 0 


Proof. From the system HJB-FP we can write 


(7.8) 




F(x, mi(s)) — F(x, m 2 (s)) 


u\{x,T) - u 2 {x,T) = F T (x,m 1 (T)) - F T {x,m 2 {T)) 


d 1 

— (mi — m 2 ) = — dw(Duimi — Du 2 m 2 ) 
ds A 


mi(x, t) — m. 2 (x, t) = m\(x) — 777 - 2 ( 3 ;) 


32 







then a simple calculation shows that 


— f (ui(x, s) — U 2 {x, s))(mi(x, s) — m 2 {x, s))dx = — f (F(x, mi(s)) — F(x, rri 2 (s)))(mi(x, s) — m 2 {x, s))dx 


ds 


* R n 


i R n 


1 


/ (mi(x, s) + m 2 {x, s))\Dui(x, s) — Du 2 {x, s)| dx 
IX J R n 


and the result follows immediately, recalling that mi, m 2 are positive and using the assumption (17.71) . ■ □ 

We now state the 


Proposition 7. We make the assumptions of Proposition^ and if 7. ,51) . 17.61) . 17.71) . We then have 


dV 

U(x, m, t) = —(m,t)(x) = u mt (x,t ) 
am 


(7.9) 


Moreover, we have the estimate 


\U(x,m,t)\ < C(1 + \x\ 2 + / |£| 2 m{f)dC) 


(7.10) 


1 R n 


Proof. We recall the definition of the value function V(m,t), see section 1531 and formulas (15.201) and (17.31) . 
Let m\{x) be some probability density and the functions u\(x,s) = u mi t(x,s), m\(x,s) = m mi t(x,s) 
solutions of the system HJB-FP (15.181) . The feedback v±(x,s) = ——Dui(x,s) is optimal for the control 
problem (15.141) . (|5.15D . (j5.16[) . The corresponding optimal trajectory, starting from a deterministic value x 
is y X mit(s). The probability density m,-^(x, s) corresponding to the feedback Di(x,s) is the image of mi by 
the map x —> y X mit(s), so we can write 


mi(s) = m {ll (s) = y mit (s)(.)(mi) 

We now consider another initial probability density 7712 ( 3 ;) and the same feedback vi. Namely we compute 
Jm, 2 t(v i(-)). The probability density at time s, with initial condition at time t equal to m -2 and feedback v\ 
is y mi t{s){.){m 2 ) denoted m\ 2 {s) = m\ 2 {x,s). It is solution of the FP equation 


dm\2 

ds 


A 


div(Hui(x)mi2) = 0 


m 12 (x, t) = m 2 (x) 


We can then write 
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Jm2t.{vi{-)) — ^ 


/ / |-D«i(x, s)| 2 mi 2 (x, s)dxds + / / mi2(x,s)f(x,mi2(s))dxds+ 

t J R n Jt J R n 


+ / mi2(x,T)h(x,mi2(T))dx 
J R n 


Therefore we have the inequality 


V(m. 2 ,t ) < J m 2 t (v i(.)) - </rmt(xi(-)) 


1 


|Z)ui(x, s)| 2 (mi 2 (x, s) — mi(x, s))dx<is+ 


2A J t J Rn 

+ / (mi 2 (x,s)/(x,mi 2 (s)) - ml(x,s)/(x,ml(s)))dx<i,S+ 

,/^ J R n 

+ / (mi 2 (x, T)h{x, mi 2 (T)) — mi(x, T)h(x, m\(T)))dx 

J R n 


We note that 


hence, as easily seen 


~ ^div(T>ui(x)(mi 2 - mi)) = 0 
m i 2 (x, f) — mi(x, f) = m 2 (x) — mi(x) 


5 1 

+ ^l-Duil 2 = f ( ;c > m i( s )) 

ui(x,T) = F t (x, m\(T)) 


f ui(x, f)(m 2 (x) — mi(x))dx = — f I |D«i(x, s)| 2 (toi 2 (x, s) — mi(x, s))dxds+ 
IR n 2A Jt J R n 


+ / F(x, mi(s))(mi 2 (x, s) — m\{x, s))dxds + / Ft(x, rai(T))(mi 2 (x, T) — mi(x, T))dx 

Jt J R n J R n 


(7.11) 
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Combining with (17.111) we can write 


V(m, 2 ,t) — V(mi,t) < / u\(x, t)(m 2 (x) — mi(x))dx+ 

J R n 

+ / [mi 2 (x,s)f(x,mi 2 (s))—mi(x,s)f(x,m,i(s))-F(x,mi(s))(mi 2 (x,s) — mi(x,s))]dxds+ 

Jt J R n 


i R n 


[m l2 {x,T)h(x,mi 2 {T)) - mi(x,T)f(x,mi(s)) - Ft(x, mi(T))(mi 2 (x,T) - mi(x,T))]dx 


Recalling that F(x,m) is the functional derivative of f Rn f(x,m)m(x)dx, we can write the above inequality 
as follows 

V(m, 2 ,t) — V(mi,t) < / u\(x, t)(m, 2 (x) — rri\(x))dx+ (7-12) 

J R n 

+ f [ f f [ 0^—(x,mi(s)+6iJi(mi2(s)—mi(s))(£,)(mi2(x,s)—mi(x,s))(mi2(£,s)—mi(£,s))dxd£dsd6diJ,+ 

Jt JR n Jr™ Jo Jo 9m 


+ 


1 rl OF 

6»—(x, mi(T)+0fi(mi2(T)-mi(T))(g)(rrii2(x, T)-m 1 (x, T))(m 12 (£, T)-mi{£, T))dxd^d9dy 



JR n JR n J0 Jo 

Recalling that mi 2 (s) = y mi t(s)(.)(m2) and mi(s) = y mi t(s)(.)(mi), we can write for a test function ip(x,£) 


/ / <p(x,£)(m 12 (x,s) - m^z, s))(m 12 (f, s) - s))dxd£ = 

J R n J R n 

/ / l p{yxmit{s),y^m 1 t{s)){m 2 (x) - mi(x))(m 2 (€) - mi(g))dxd£ 

J R n J R n 

We introduce a pair of random variables X\, X 2 whose marginals are mi, m 2 -We then introduce an inde¬ 
pendent copy Yi, I 2 . It is easy to convince oneself that we have the relation 


I R n J R n 


‘P(yxmit(s),y^m 1 t(s))(m2(x) - mi(x))(m 2 ($,) -mi(g))dxd 4 = 


n F D^D x ip(^yXimit{s) + 0(yx2mit(s) yX\mit{,s)\yY\m\t{,s) F n(yY 2 mit(s) VYimiti^))) 

(yX2mit(s) — yXimit(s))('!/Y2mit(s) yYimit(.s)')ddd[X 

If we have \\D^D x (p(x,^)\\ < (7,then we get 
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I R n J R n 


v{yxmit{s),ytmit{s))(m, 2 (x) - mi(x))(m 2 (Q - mi(£))dxd£\ < 


CE\yx 2 mit (s) yXimit{ s ) I \yY2rr1\t ( s ) yYimit(s) 


and from the independence property 


— C(E\yx 2 mit{ s ) yXimit( s ) I)"" 5: CE\yx 2 mit{ s ) yX\m\t{.s)\ 


Using property (16.161) we obtain also 


lR n JR 


/ <p(yxm 1 t(s),yz mit {s)){m 2 {x) - mi(.x))(?n 2 (0 - mi(€))dxd£\ < CE\X 2 
jR n 


-Xi| 


and sinceA'i, X 2 have an arbitrary correlation, this implies also 


1 R n J R n 


‘p(yxm 1 t(s),y^mit(s))(m 2 (x) - mi(z))(m 2 (£) - mi(g))dxd4\ < CW%(m 1 ,m 2 ) 


We may apply this result with <p(x,£) = ^(r,mi(s) + Ofi(mi 2 (s) — mi(s))(£). Thanks to assumption (17.61) 
the same result carries over. Therefore we conclude easily the estimate 


V(m, 2 ,t) — V(mi,t) < / u\(x, t)(m 2 (x) — m\{x))dx + CW$(mi, m 2 ) 

J R n 


(7.13) 


Interchanging the role of mi, m 2 , we have also 


U(mi, t) — V(m 2 , t) < / u 2 (x, t){m\(x) — rri 2 {x))dx + CWf(mi, m 2 ) 

< f M,,t) (mi (x) -™ 2 (x)Mx + / („(,,*) -„(*„()(„,,(*)- m2 ( X )) dx + CW^um 2 ) 
J R n J R n 

and from Proposition [U] and assumption (17.71) the 2nd integral is negative, which implies 

V(m u t)-V(m 2 ,t) < [ u 1 (®,i)(m 1 (s) - m 2 (x))dx + CWi(m 1 ,m 2 ) 

J R n 

or 


V (m 2 , t) — V(mi, t) > [ ui(x,t)(m 2 (x)-mi(x))dx-CW 2 2 (mi,m. 2 ) 

JR n 
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and comparing with (17.131) we can assert 


\V(m, 2 ,t) — V{m\,t) — / ui(x,t)(m 2 {x) — m\(x))dx\ < CW$(mi,m, 2 ) (7-14) 

Jr n 


Now we have 


\u m t{x,t)\ < C(1 + \x\ 2 + [ |£| 2 m(C)di (7.15) 

J R n 

So for any curve m e € V 2 , u m t(x,t) € L ] mr . From the estimate (17.141) we get immediately the result (17.91) . 
The proof has been completed. ■ □ 

7.2 OBTAINING BELLMAN EQUATION 

We have seen in section I6.2t hat 


and thus 


z xrnt( J) — Du m t{y X mtJs) > &) 


D x U(x,m,t) = z xmt (t) (7.16) 

Therefore from the estimate (16.171) we can assert that 

J l£IM£R] (7.17) 

In particular, we can see that D x U(x,m,t) belongs to L^(i? n ; R n ). But then, recalling the correspondance 
V(X,t) = V(m,t)\ m= c x ,we can write 


\D x U(x,m,t)\ < C[1 + |x| + 


D x V(X,t ) = D x U(X,C x ,t) (7.18) 

and 

\\D x V(X,t)\\ 2 = [ 1 D x ^ (x)\ 2 m{x)dx 

J R n om 

If we look at the Bellman equation in the Hilbert space TL, see (12.111) we obtain exactly (I5.21|) . So we can 
state 

Proposition 8. We make the assumptions of Proposition Q The value function V(m,t) of the problem 
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\5.1A\) J5.17\) or equivalently \5.15\) . 115. 1(A) satisfies the estimates i7.2l) . \7.1 ill ) (with U(x,m,t ) = -—^—— (x) 
1 - dm 

) and /1 7. 171) . It is the unique solution, satisfying these estimates, of the Bellman equation \5.21\) . Moreover, 

we have the explicit formula 17,11) with y xm t(s) being the unique solution of \ 6. 10\) and z xrn As) = —A ^ 

ds 


7.3 OBTAINING THE SCALAR MASTER EQUATION 

We can derive the scalar master equation from the probabilistic master equation which we write as 

follows 


^ - ±.D x \\U(X,t)\\ 2 + D X F{X) = 0 (7.19) 

U{X,T) = D x F t {X) 

We know that U(X,t ) = D x U(X, CxA) and U(x,m,t) = u m t(x,t) and D x ii mt (x,t) = z xm t(t). Therefore 
||7/(A,t)|| 2 = J Rn m, t)| 2 m(£)d£. Since this functional of m only has a derivative in the Hilbert space 

B it can be written as follows 

A p 

D x \\U{X,t) || 2 = D x — (J \D^U^,m,t)\ 2 m^)df)(X) 

Recalling that DxX(X) = D x F(X,m), DxXt(X) = D x FT(X,m ), we see that (17.191) can be wriiten as 
follows 


D ^dUiXrnX[ _ l__d_ ( j lDeU((tmtt)] 2 m(m)(x) + F(x ,m)] = 0 

D x U{X,m,T ) =D x F T (X,m) 


This leads to 


dU(x, m, t ) 
dt 


J__d_ 
2 A dm 


( [ \D i U{f,m,t)\ 2 m{f ! )dC){.x)+F{x,m) = 0 
J R n 


U ( x , m, T) = Ft(x, m) 


d d 

which we can write as fl5.22j) . taking account of the symmetry property —— U ( x , m, f)(£) = —— U (£, m, t)(x) 
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o 

This proof is not fully rigorous. It assumes implicitly the existence of —— U(x, m , f)(£), which is the 2nd 

dm 

d 

derivative of the function V(m,t). To study it rigorously and give an implicit formula for —— U(x,m,t){£) 

dm 

, one can use the system of HJB-FP equations (15.181) and write the solution as u m t(x,s),m m t(x,s) to em- 

du m t(x, s) dm mt (x,s) 


phasize the initial conditions m, t. We then consider the functional derivatives 


-( 0 , 


(0 


dm dm 

and differentiate formally the system of HJB-FP equations. To simplify notation, we take a test function 


m(£) and consider 


which we note u m t-m(x,s), rh m t-m(x,s) and to simplify further u(x,s), m(x,s). In particular , u(x,t) = 
d 

f —— U(x, m, £)(£)m(£) d£. The pair u(x, s),m(x, s ) is solution of a system of linear P.D.E. as follows 



x,m(s))(£)m(£,s)d£ 

(s,m(T))(0m(e,T)de 


(7.20) 


dm 

ds 


A 


div(Z>« rri + Du m) = 0 


rh(x, t) = rh{x) 


This system is obtained by linearization of the system (15.181) . The functions u(x, s),m(x, s ) are solutions of 
the system (15.181) . We can write also 


u(x,s) = Js ^(y X mt(v),ymt((7)(-)(rn))(t)m(£,a)d£da+ 


R n 


dF T 

dm 


(yxmt{T),y mt {T)(.)(m))(£)m(£,T)d(; 

(7.21) 


We can then study (17.211) as a fixed point equation in the function u(x, s ). 


8 QUADRATIC CASE 

8.1 ASSUMPTIONS 

We consider the quadratic case, {HID- We also take 
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( 8 . 1 ) 


h(x,m ) = ^(x — Stx)*Qt(x - Srx) + ^ x*Qtx 


In the space T-L we have 


-F(X) = ^EX*(Q + Q)X + ^£W*(S*QS - QS - S*Q)EX 
X T (X) = X -EX\Q t + Q)X + ^EX*{S*QS - QS - S*Q)EX 


( 8 . 2 ) 


We can write E{X) = Ef(X,Cx) with 


f(x, m ) = i(x — Sx)*Q(x — Sx ) + ^x*Qx 

in which x = f xm(x)dx, assuming the probability law of X has a density, m. So we can also write 


(8.3) 


F(X) = <f>(m) = 7 : [ x * 

2 jRn 


(Q + Q)xm{x)dx + - f x*m(x)dx (S*QS — QS — S*Q) f xm(x)dx 
2 ./ R n J R n 


I R n 


f(x, m)m{x)dx 


If we take m € L 2 (R n ) n L l (R n ) not necessarily a probability density, then we have to introduce m 1 
f Rn m(x)dx and write 


$(m) = / /(x, m)m(x)dx = 

J R n 


(8.4) 


= — / x * (Q + Q)xm(x)dx + -x*S*QSx mi — -x*(QS + S*Q)x 
2 J Rn 2 2 

ckE>(m) 

We have noted F(x,m) = —--(x). Then as a Gateaux differential we have 

dm 


F(x, m ) = -x * (Q + Q)x + x*(S*QS mi — QS — S*Q)x + -x*S*QSx 


(8.5) 


We note that 


D x E(X) = (Q + Q)X + (S*QS - QS - S*Q)£X 


( 8 . 6 ) 


So the equality DxX(X) = D x F(X,m) is true only when mi = 1. It is important to keep in mind that 
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when we work with Gateaux differentials, we have to make calculations with the term mi,even though that 
eventually, when applied to m = probability density, we shall have mi = 1. To understand further this 
point, let us compute the 2nd derivative. We have 

dF - - 

— (®,m)(£) =x*S*QS(x + 0+?(S*QSm 1 -QS-S*Q)x (8.7) 

am 

We see that this formula is symmetric in x. £ as needed. Without the term mi in (18.5|) this will not be true. 
We have 


D l F {x,m ) 


OF 

Q + Q, D x D^ — (x,m)(Q 


(S*QS mi — QS — S*Q) 


and , see [5] 


D\F(X)Z = D 2 x F(X,Cx)Z + EyzD x D y —(X,£ x ){Y)Z = 
= (Q + Q)Z + ( S*QS -QS- S*Q)EZ 

which is exactly what we obtain by differentiating (18.61) in the Hilbert space. 

8.2 BELLMAN EQUATION 

Bellman equation ( I5.21|) writes 


dV 

~dt 



D dV(m,t) 
^ dm . 


(01 2? ™(£)d£+ 


+ 7 : [ £ * (Q + <2)£m(£)d£ + \-x*S*QSxm\ 

1 J R n 2 


^x*(QS + S*Q)x = 0 


( 8 . 8 ) 


V(m, T) = 


If 

z Jr " 


£ * (Qt + QT)£ra(£)d£ + \x*SyQxSTxmi — \x*{QtSt + S r Qt)x 


The solution is 


V (m, 



£ * P(t)£m(£)d£ + - x*T,(t ; mi)x 


(8.9) 


with 
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( 8 . 10 ) 


dP _ 
dt 

P(T) 


P 2 

~r -h Q + Q = 0 

A 


= Qt + Qt 


1 1 

— - T ( SP + PS ) - T s2m i + 5*05mi - (QS + S*Q) = 0 
dt A A 

S(T; mi) = S^QtSt m i — (QtSt + S^Qt) 


( 8 . 11 ) 


8.3 MASTER EQUATION 

The scalar Master equation (15.221) reads 


or r i /» orr i 

— d £—( x,m,t)(£).D(:U(£,m,t)m(t)d$, - — \D x U(x,m,t)\ 2 
+^x *(Q + Q)x + x*(S*QSmi - QS - S*Q)x + i x*S*QScc = 0 
U(x, m, T ) = ^x * (Qt + Ot)^ + x*(StQtSt mi - Qt5t - S^Qt)x + ^ x*StQtStx 


( 8 . 12 ) 


Its solution is 


TT , . dV{m,t) . . 1 . 1 _*<9E(t; mi) _ 

U(x,m,t) = —---(x) = -x P(t)x + x Pit] mi)x H—x —---x 

am 2 2 ami 


(8.13) 


We have 


U X C/(x, m, i) = P(t)x + £(i; mi)x 


(8.14) 


0*7, .a ft\ -*^S(t;mi) , ^ ^ , 1 _* d 2 H(t; mi) _ 

S^{ ’*>® = X + i)+i E(<; mi)l + 2* Dm* X 




dY*(t\ mi) 
dm i 


x + E(f; mi)x 


We note that T(t; mi) 


<9E(i; mi) 
dm i 


satisfies the equation 
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(8.15) 


^ - \(T(P + Smi) + (P + Emi)r) - htf + S*QS = 0 
at X A z 

r(T;mi) = S* t Q t St 

and we check easily that the function U(x,m,t ) defined by (18.131) is solution of the scalar master equation 

{ffl). 

We turn to the vector master equation (I5.25|) which reads 


— D£—(x,m,t)(£)U(£,m,t)m(€)d£-jDJA(x,m,t)l((x,m,t)+ (8.16) 

+(Q + Q)x + ( S*QSm\ — QS — S*Q)x = 0 

U(x, m, T ) = (Q t + Qt)x + (S^ - QtSt ~ StQt)x 

whose solution is 


U(x, m, t ) = D x ll(x, m, t) = P(t)x + S(t; m\)x (8-17) 

This statement is easily verified. 

8.4 SYSTEM OF HJB-FP EQUATIONS 

We now look at the system (15.181) which reads 


-1T + ^\ Du \ 2 = \ X *(Q + Q)x + ®*(s)(S*QSmi(s) - QS - S*Q)x + \x*(s)S*QSx(s) (8.18) 

os Z\ Z Z 

u(x,T) = 1 x * (Q t + Q t )x + x*(T)(S^Q T S T m 1 (T) - Q T S T - S* T Q T )x + h* (T) S^Q t S t x(T) 

dm 1 

—- -div(Dum) = 0 

os A 

m(x, t) = m(x) 

It is immediate to see that mi(s) = mi.The function x(s) represents the mean f Rn ^m(^, s)d^. We do not 
need to obtain the full probability m(x, s).The mean is sufficient. One can check the formula 
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(8.19) 


In particular u(x, t ) 
follows 


/ \ ^ L *yV \ — / \ , ^ — ( \* dYj(s 7 TYl 1) _ . . 

u(x,s) = -x Fls)x + x 2 j(s; mi)x(s) H— xls) --- xls) 

2 2 ami 

= U(x,m,t) given by (18.131) . Also u(x,s) = U(x,m(s),s). Note that x(s) evolves as 


(It 1 

YT + t(P(s) + miS(s; mi))®(s) = 0 
as A 


x(t) = x 


( 8 . 20 ) 


We have seen that U(x,m,t) is differentiable in m with 


du , 4.\(t\ _*3E(t;m 1 ) < , , ^ , , 1 _* d 2 Y>{t] mi) _ 

—{x. m. 0(5) = x —(X + 5) + 5 S(t; mi)x + 5 X x 


If we consider a test function m(£) and define 


«(x, t) 


u mt -m(x,t) = lim 
0— 


’U"m+6m,t{-E j ^ mt(x, t ) 

0 


we get 


-5 xi *wx ^ , -*3S(t;m 1 ) = , _ 

ilx, t) = x zdr; mj h + x —-- x + m i(x 

ami 




dm 


l 


X+-X 


* ^ ^ V? " •'l / 


dm.f 


x) 


( 8 . 21 ) 


We can also compute 


u(x,s) = U mt -rh{x,S) = J™ 

6 —^0 


'^m+ 6 m,t{p^i ^) u m,t (? ®) 

0 


We have 


u(x, s) = x*(S(s; mi)x(s) + 


<9£(s; mi) 
dm. i 


x(s)rhi) + ®*(s 


9S(s; mi) 
dm. i 


®(s) + fh\—x*(s) 


1-*, ,<9 2 X(s;mi) 


dm\ 


x(s) (8.22) 


in which 


dx(s) 

ds 

and where mi = 


+ y(-F*(s) + ?«iS(s; mi))x(s) + Y?hi(£(s; mi) + mi ^ S ’ mi ^ )a:(g) = 0 (8.23) 

A A <7777-1 

m(£)d£. The function u(x,s) is the solution of the linearized equation (17.201) . namely 
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r\~ -i 

— ——b —Du.(P(s)x + S(s; mi)x(s)) = x* (s*QS x(s)mi + (S*QS mn i — QS — S*Q)x(s)'\ + (8.24) 

as X v / 

+x(s)*S*QSf(s) 


u(x, T) = x* ( S*QSx(T)fh i + (S*QSmi - QS - 5*Q)x(T)) + 
+x(T)*S*QSl(T) 


which can be checked by direct calculation. 


8.5 STATE EQUATION 


We consider equation (16.101) in the quadratic case. Since we know the function u m t(x , s) see (I8.19|) the best 


is to use the fact that y xm t(s) is the solution of 


y{t) = x 


We get the explicit solution 


l r 8 i r s i r s 

yxmt{s) = exp-jJ P(a)dcr x — — J (exp—— y P(r)dr £(<r) exp 


1 

A 



(£(r) + P(r))dT)da x\ (8.25) 


8.6 FORMULATION IN THE HILBERT SPACE 


We can formulate Bellman equation and the Master equation in the Hilbert space 7~L . We have first Bellman 


equation 


% - ^\\D X V\\ 2 + ]-EX*(Q + Q)X + \eX*{S*QS - QS - S*Q)EX = 0 (8.26) 

ot z\ z z 

V{X,T) = i EX*{Q t + Qt)X + ^EX*(S* t Q t St - QtS t ~ S^Q T )EX 


whose solution is 
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(8.27) 


V(X,t) = i EX*P(t)X + ^EX*Y(t)EX 
with S (t) = T,(t; 1). The Master equation reads 

^ - \d x U(X, t ) U(X, t) + {Q + Q)X + ( S*QS - QS - S*Q)EX = 0 (8.28) 

ot A 

U(X, T) = ( Qt + Qt)X + {S^QtSt - QtSt - S* T Q T )EX 

whose solution is U(X,t ) = P(t)X + T,(t)EX. Note that D x U(X,t)Z = P(t)Z + S [t)EZ . The state 
equation is the solution of 

( ]Y 1 

— = -~(P(8)Y(8) + E(s)EY(s)) (8.29) 

Y(t ) = X 

hence the formula 


Y (s) = exp — 


P(a)daX- 


~ / (exp-- 


1 


P{r)dT S(cr) exp — — / (P(r) + Yi{r))dT)da)EX 

A It 


(8.30) 
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